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ABSTRACT 

Principles are established for reaching valid 
decisions as to whether evaluative judgments should be nora 
referenced or criterion referenced. A set of decision rules is 
presented, in flow chart form, to serve as a guide to the 
decision-making process. The use of these decision rules is 
illustrated for a variety of practical situations. The evaiuati ve 
framework is founded upon credible principles of teaching and a 
precise language of curriculum and instruction. These bases include 
the pragmatic, conceptual, and value aspects of education. (RC) 
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INTRODUCTION 


The Problem And Its Significance 


In many practical evaluative situations, a decision must 
be made as to whether an evaluative judgment of pupil performance 
should be a criterion--referenced judgment or a norm-referenced 
judgment. The question with which this paper is concerned is: 
How cana justified decision be made as to which is the appro- 


prtate frame of reference? 


The authors' intention is to establish credible prin- 
ciples for reaching, in every relevant decision-situation, a 
valid decision as to whether an evaluative judgment of pupil per- 


formance should be norm-referenced or criterion-referenced. 


It is believed that the decision rules developed here can 
provide sound bases for practical judgments in an area where 
judgments are ordinarily made in the absence of appeal to cre- 
dible principles. As an auxiliary outcome, the complete range 
of pupil evaluations of relevance to the curricular and in- 


structional enterprise will be delineated. 


The Organization Of This Paper 


Achievement of the stated purpose will be accomplished 
through the following course of action. First, concepts of crite- 


rion-referenced tests and norm-referenced tests will be clarified; 
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and the concepts of criterion-referenced judgments and norm- 


referenced judgments will be clarified. 


Second, the nature of a curricular framewcrk for evalua- 
tive judgmentswill] be described, and significant interrelation- 
ships within that framework will be identified. Also, the 
nature of a curricular-free framework for evaluative judgments 
will be described. The range of evaluative judgments in refer- 
ence to pupil performance will.:be delineated for each type of 


framework. 


Third, the logic of the decision as to whether a judgment 
should be criterion-referenced or norm-referenced will be set 
forth, and its utilization illustrated for all possible decision 


situations. 


Fourth, some concluding statements wil? be made to illum- 


inate or extend fundamental aspects of the outlined principles. 


THE CONCEPTS OF CRITERION-REFERENCED AND NORM-REFERENCED 


Criterton-Referenced Tests And Norm-Referenced Tests 


The primary difference between criterion-referenced tests 
and norm-referenced tests lies with the manner in which the tests 
are constructed (i.e. the construction of the test items). Cri- 
terion-referenced tests consist of items, each of which is log- 
ically valid in reference to some set of concepts or statements. 
For example, the test items may be constructed in reference to 


a set of explicitly stated objectives. 


Norm-referenced tests consist of items selected from a pool 
of items on the basis of item statistics obtained by item try- 
outs on relevant groups of students. Such tests generally con- 
tain items which require performance beyond, as well as below, 
the expected level of performance for the intended population. 
The items are constructed and selected so as to m»ximize the 
range of scores made by a given group of students; hence, they 
represent gradations in knowledge or skills relevant to the 


instructional unit. 


A test is used as a device to elicit relevant pupil per- 
formances. A set of sentences describing the way a pupil per- 
forms on a test may appropriately be viewed as a report of the 
observed performance of that pupil. The process of making eval- 
uative judgments about such reports of observed pupil performance 


is the focal point of the following section. 


5 


Criterton~Referenced Judgments And Norm-Referenced Judgments 


The primary difference between criterion-referenced judg- 
ments of pupil performance and norm-referenced judgments lies 
with tia: bases in which the judgments are grounded. A criterion- 
referenced judgment of the performance of a pupil is made by com- 
paring reports of the observed performance of that pupil against 
a set of statements from which the desired manner of performance 
can be derived. That set of statements, or some set of derived 
statements, constitute the "criteria" which form the frame of 
reference for the judgments; hence the label "criterion-ref- 


erenced judgments." 


A norm-referenced judgment of the performance of a pupil 
is made by comparing reports of the observed performance of that 
pupil against a set of observation reports of the performances 
of other pupils. The set of observation reports of the per- 
formances of other pupils constitute the "norms" which form one 
aspect of the frame of reference for the judgments; hence the 
label "norm-referenced" judgments. Of course, no judgment of 
the performance of a pupil can be made in reference to norms 
without appeal to values, ethical rules, or some kind of 


principles as the grounds for judging the relationship between 


the individual performance and the "norm;" in this sense the 
use of criteria in making evaluative judgments can never be 


evaded. 


CURRICULAR AND CURRICULAR-FREE FRAMEWORKS 
FOR EVALUATIVE JUDGMENTS 


The Nature Of A Curricular Framework For Evaluative Judgments 


The Structure Of A Curricular Framework 


Adequate curricular representation requires articulation 
of goals, procedural rules, and qualifying conditions. Curric- 
ular goals represent the ends to be achieved by the students. 
Curricular procedural rules represent the means by which those 
ends are to be achieved. They specify the basic substantive or 
content elements with which instructors and students will deal; 
and the manner in which the instructors and students will deal 
with those substantive elements and the way they will relate to 
each other as well. The qualifying conditions represent student 
characteristics believed necessary in order to achieve the goals, 


through implementation of the procedural rules. 


The relationship between a goal-component (ends), a rule- 
component (means), and a qualifier-component (conditions under 
which the means are believed effective for achieving the ends) 
constitute a claim. This kind of claim will be called a curricular 


claim; and its generalized form may be represented as follows: 


Generalized Curr. cular Claim 


For each puptl X, where X satisfies conditions C; 
If both the teacher(s) and a set of pupils, 
of which X is a member, act under rules R, 
then X will (probably) attain goal-state S. 


rhe ends, means, and qualifier components of a curricular 
claim will be called, respectively: curricular goal-state, 


curricular rules, and curricular qualifying conditions. 


The curricular rules leave both teacher and pupil some 
degrees of freedom to determine the way in which their individual 
acts will be performed. The teacher develops strategies for 
influencing the pupil toward goal-state attainment, and adapts 
these strategies in response to the patterns of individual 
pupil acts. Of course a given strategy is reasonable only if the 
teacher believes that, under the existing circumstances, implemen- 
tation of that strategy will lead to achievement of certain ends. 
This entails the requirement that a teacher's strategy must be 
imbedded within the rule component of some claim that the teacher 
has reason to believe is valid. This sort of a claim will be 
called an instructional claim* whose generalized form may be 


represented in the following way: 


Generalized Instructional Claim 
For each puptl X, where X satisfies conditions I, 
If the teacher(s) acts under rules T, in relatton 
to a set of puptls of which X ts a member, 
then X will (probably) attain goal-state G. 
The ends, means, and qualifier components of an instruction- 


al claim will be called, respectively: instructional goal-state, 


instructional rules, and instructional qualifying conditions. 


* The concepts of "curricular claim" and "instructional claim" are discussed 
further in "The Validation Of Educational Programs," by Emajean McCray 
and John Lottes, a paper presented at the 1976 AERA conference. 
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The Range Of Evaluative Judgments 
In Reference To A Currteular Framework 


Under the preceding curricular framework, the types of 
evaluative judgments to be made can be categorized as follows: 
(1) judgments of student performance in reference to the curric- 
ular qualifying conditions, (2) judgments of student performance 
in reference to the curricular rules, (3) judgments of student 
performance in reference to the curricular goal-state, (4) judg- 
ments of student performance in reference to the instructional 
qualifying conditions, and (5) judgments of student performance 


in reference to the instructional goal-state. 


The Nature Of A Curricular-Free Framework For Evaluative Judgments 


The Meaning Of Curricular-Free Framework 


All judgments of student performance can be described as 
belonging to a universe of evaluative judgments made in reference 
to student performance. One class of judgments of student per- 
formance was described in the previous section; namely, judgments 
in reference to a curricular framework. All judgments of student 
performance, which are not made in reference to a curricular 
framework, are made in reference to some other framework which 
is grounded outside the curricular framework. All such extra- 
curricular frameworks for evaluative judgments will be called 


ecurrteular-free frameworks. 


‘ The Functions Of Curricular-Free Frameworks 


Judgments governed by curricular-free evaluative frame- 
works are often made about student performances. These judgments 
are of varying kinds and can be categorized loosely as follows: 
(1) assessment of pupil performance to determine educational 
institutional status and (2) assessment of pupil performance 


to determine individual status. 


Assessment of pupil performance to determine educational 
institutional status may be conducted in reference to a school, 


district, state, regional or national level educational system. 


This category would include the assessment of the strengths 
and weaknesses of a particular school or of the schools in a 
particular geographic area; as well as the assessment as to 


whether a particular school or group of schools is effective 


in influencing student attainment of institutional goals. 


Assessment of pupil performance to determine individual 
status would include (a) the determination as to whether an 
individual attained institutional goals and (b) the determina- 
tion of individual strengths and weaknesses. The latter aspect 
would include the diagnosis of individual learning problems 
which is a type of judgment often conducted outside the curric- 
ular framework. Such diagnosis might yield evaluative judg- 
ments as to physical defects, sensory processing weaknesses, 


psychological abnormalities, and so forth. 
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The ‘determination of individual strengths and weaknesses 
would also include judgments made in the selection of personnel 
(or students). This process could involve the selection of 
students for entry into college, pdaaeen programs (i.e. special 
education, vocational education, programs for the gifted, etc.) 
or particular jobs based upon student performance which is 


curricular-free. 
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CRITERION-REFERENCED OR NORM-REFERENCED JUDGMENTS: 
THE LOGIC OF THE DECISION 


The decision rules for determining whether a judgment 


should be criterion-referenced or norm-referenced are shown 


in CHART A. These decision rules provide the basis for making 


rational decisions concerning a wide range of practical educa- 


tional situations. 


Evaluation within the curricular-instructional claim frame- 
work involves judgments which are made against specific criteria. 
This situation results from the fact that all activity in the 
curricular-instructional realm is goal directed and governed by 


rules which apply only when specified guidelines or conditions 


are satisfied. 


er 


Diagnosis of an Gea dues may be conducted within the 
curricular framework or within a curricular-free framework. 
When diagnosis occurs within a curricular framework, the evalu- 
ative judgments are criterion-referenced, i.e. they are based 


upon specific criteria given by some component of the curricular 


framework. 


Diagnosis of individual pupil performance in the curricular- 
free framework should be included in the assessment of individual 
strengths and weaknesses (see CHART A), and should require a 


criterion-referenced judgment. This same category (i.e. "To 
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CRITERION- OR NORM-REFERENCED EVALUATIVE JUDGMENTS 
THE LOGIC OF: THE DECISION 
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* An “educational institution” may have school, 


district, state, regional or national scope. 


CHART A 


yes 
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determine individual strengths and weaknesses") also includes 
judgments made about individuals for the purpose of counseling 
or job selection. These judgments could be either criterion- 
referenced or norm-referenced depending upon the purpose. For 
example, a student might be selected for a particular job be- 
cause of performance judged in reference to a job analysis 
(i.e. criterion-referenced judgment); whereas in another case 
a student might be selected for a particular job based on judg- 
ments of performance relative to that of other students (i.e. 


norm-referenced judgment). 
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CONCLUDING STATEMENTS 


This paper has described the nature of a curricular 
framework as well as that of a curricular-free framework for 
evaluative judgments of student performance. The range of 
evaluative judgments in reference to pupil performance was de- 
lineated for each type of framework. The logic of the decision 
as to whether a judgment should be criterion~-referenced or 
norm-referenced was set forth in flowchart form; and included 


a wide range of practical decisions encountered by educators. 


Evaluative judgments within the curricular-free framework 
can be either criterion-referenced or norm-referenced. However, 
there are no intra-curricular judgments which may be norm- 
referenced. All evaluative judgments of student performance 
within the curricular framework are criterion-referenced because 
they are based upon specific criteria given by some component 


housed within the context of a curricular claim. 
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